Skip to content

refactor: execute Planner precompute DAGs for raw heaps - #800

Open
zzylol wants to merge 3 commits into
refactor/precompute-dag-backfill-revisionsfrom
refactor/precompute-dag-heaps
Open

zzylol wants to merge 3 commits into
refactor/precompute-dag-backfill-revisionsfrom
refactor/precompute-dag-heaps

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #797.

Why

After #796/#797 every raw output except CMS/CountSketch heaps executes Planner's precompute graph. Heaps stayed on the backend's per-sample kernel (the last item/weight interpreter in raw_dag.rs) because stored heap readout could only decode the legacy integer heap kernel, not Planner's weighted-frequency state.

What

  • Storage (backend-owned format): a Planner heap window is stored as a WeightedFrequencyV1 frame (sketchlib WeightedFrequency bytes, persisted with encoding tag 6). One helper picks a frame's encoding from its state for live appends, revision views and derived publications; carry-in checks treat it as a full frame.
  • Readout: SummaryState::WeightedFrequency decodes such frames for their catalog family and matrix shape, merges them, and ranks items under the legacy heap key (;-joined parts; a canonical series identity renders as its series key). PointCount/total over these frames is reported unsupported.
  • Execution: heaps build through the Planner graph like every other raw output; the backend item/weight interpreter is deleted. A heap's empty state is a Planner heap, so revisions install for heap outputs.

Before this PR

topk(3, count_over_time(top_endpoint_qps[5s])) with a CMS heap: each sample was keyed by the backend (series key string) and added to a CountMinSketchWithHeapAccumulator.

After this PR

The same output is built by Planner's KeyedSummaryBuild over the pane batch (item = LabelIdentity of the series' label set); registered_temporal_topk_cms_heap / _count_sketch_heap return the same ranked items (top_endpoint_qps{endpoint="gamma"} = 200, …) and range results as before.

Behaviour differences

  • Heap scores are exact Float64 weights; the legacy CMS heap rounded each weight to an integer count.
  • New heap windows are WeightedFrequencyV1 frames; a range mixing them with legacy heap frames of the same output fails with a family mismatch (a new plan generation starts new outputs). Older binaries do not read tag 6.
  • Heap items that exclude identity labels (topk by (…) identity items) are rejected at install until readout can restore their group labels.

Remaining (backend computation still present)

  • RawDagProgram::empty_state constructs empty family states (heaps via Planner's constructor; other families via the backend kernel factory) for known-empty counter panes and the revision codec check.
  • PrecomputeMaterialization computation fields (aggregation_type, parameters, spatial_filter, aggregated_labels, sample_update_rule) are still cross-checked against the DAG at install; removing them changes plan identities.
  • OTLP-delivered sketch panes are still merged with the kernel's merge_with, and late admitted corrections are coalesced with merge_with (both surface errors).

Validation

cargo fmt --all -- --check; cargo clippy --workspace --all-targets --locked -- -D warnings; cargo test --workspace --locked --lib; cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1 (26 passed, incl. both registered heap TopK tests); cargo test --workspace --locked --no-fail-fast (every test target) passes at the top of the stack. New tests: heap frame decode/merge/rank and shape rejection, heap empty state is a Planner heap frame. Reviewed by a separate reviewer agent; findings addressed in the last commit.

🤖 Generated with Claude Code

zzylol and others added 3 commits September 30, 2026 08:11
A raw heap output built by Planner is stored as a WeightedFrequencyV1
frame (sketchlib WeightedFrequency bytes, persisted with its own encoding
tag). Stored heap readout decodes such frames for their catalog family,
merges them, and ranks items under the legacy heap key (a canonical label
identity renders as its series key). Point-count readouts over these
frames are reported unsupported rather than guessed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CMS/CountSketch heaps now take the same Planner execution as every other
raw output, so the backend's per-sample item and weight interpreter is
deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpty state

Revision views and derived publications choose a stored frame's encoding
from its state, so a Planner heap is never re-tagged as a legacy msgpack
heap; carry-in checks treat it as a full frame. Heap frames decode only for
their catalog's matrix shape, identity items render as series keys only
when they name `__name__`, and a heap's empty state is a Planner heap, so
revisions install for heap outputs. Heap items that omit identity labels
are rejected at install until readout can name their series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant